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INTERVENTION BRIEF 


Supporting Postsecondary Success l e S$ EDUCATION SCIENCES 


Open Learning Initiative (OLI) 


Online instruction, including online instruction that is blended with traditional in-person instruction, offers students more 
flexibility in the timing of their coursework and may deepen students’ understanding of the material. By leveraging online 
courseware, colleges have the potential to expand college access, reduce costs, recruit a more diverse student population, and 
accelerate instruction under some conditions.! 


Carnegie Mellon University’s (CMU's) Open Learning Initiative (OLI) provides online courses and learning materials to instructors 
and learners at low or no cost that can be presented in blended or purely online formats. OL/ courses feature learning activities 
with immediate feedback for students and a dashboard for instructors to monitor student progress. 


This What Works Clearinghouse (WWC) report, part of the WWC's Supporting Postsecondary Success topic area, explores the 
effects of OL! on course completion rates and academic achievement. The WWC identified 12 studies of OL/, three of which 
meet WWC standards.’ The evidence presented in this report includes studies of the impacts of OL! on community college 
students, including students who were White, Black, Hispanic, and Asian or Pacific Islander. 


What Happens When Students Participate in OL/?? 


The evidence indicates that implementing OLI: 
* may increase credit accumulation and persistence 


¢ has inconsistent effects on academic achievement 


Findings on OLI from three studies that meet WWC standards are shown in Table 1. For each student outcome reviewed by the 
WWC, an effectiveness rating, the improvement index, and the number of studies and students that contributed to the findings 
are presented. The improvement index is a measure of the intervention’s effect on an outcome. It can be interpreted as the 
expected change in percentile rank for an average comparison group student if that student had received the intervention. 
Findings on credit accumulation are based on one study with 605 students. Findings on academic achievement are based on 
three studies with 795 students. See Box 1 for a description of WWC effectiveness ratings. 


Table 1. Summary of findings on OL/ from studies that meet WWC standards 


Evidence meeting 
Study findings WWC standards (version 3.0) 
Improvement index Number of Number of 
Outcome Effectiveness rating (percentile points) studies students 
Credit accumulation and persistence Potentially positive effects +/ ] 605 
Academic achievement ixed effects +8 3 795 


Table Note: The improvement index is a measure of the effect of the intervention. The improvement index can be interpreted as the expected change in percentile rank for an average comparison 
group student if that student had received the intervention. For example, an improvement index of +7 means that the expected percentile rank of the average comparison group student would 
increase by 7 points if they received OL/. The improvement index values are generated by averaging findings from the outcome analyses that meet WWC standards, as reported by Bowen et al. 
(2014); Lovett, Meyer, & Thille (2008); and Schunn & Patchan (2009). Outcomes include completing and passing a course (credit accumulation and persistence) and final exam and standardized 
test scores (academic achievement). The effects of OL/ are not known for other outcomes within the Supporting Postsecondary Success topic area, including college access and enrollment; college 
attendance; college degree attainment; and labor market outcomes. 
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BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria that the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 3.0) and the Supporting Postsecondary Success topic area protocol (version 3.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and how many 
studies tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported results and about 
what will happen if the same intervention is implemented again. The following key provides a link between effectiveness ratings and 
the statements used in this report: 


Effectiveness Rating Rating Interpretation Description of the Evidence 


Positive (or Negative) Effects The intervention is likely to change an Strong evidence of a positive effect, with 
outcome no overriding contrary evidence 


Potentially Positive (or Negative) Effects § The intervention may change an outcome Evidence of a positive effect with no 
overriding contrary evidence 


No Discernible Effects The intervention may result in little to no No affirmative evidence of effects 
change in an outcome 


Mixed Effects The intervention has inconsistent effects Evidence includes studies in at least two 
on an outcome of these categories: studies with positive 
effects, studies with negative effects, or 
more studies with indeterminate effects 
than with positive or negative effects 


How is OL/ Implemented? 


The following section provides details of how OL! was implemented. This information can help educators identify the 
requirements for implementing OL/ and determine whether those implementation requirements would be feasible at their 
institutions. Information on OLI presented in this section comes from the three studies that meet WWC evidence standards 
(Bowen et al. 2014; Lovett, Meyer, & Thille, 2008; Schunn & Patchan, 2009), from OLI's website, and from correspondence with 
the developer. 


* Goal: OL! is designed to support learning and instruction via the high-quality online courses and learning materials available 
at low or no cost available on the OL/ website. 


¢ Target Population: Most OL/ courses are open to both students who take credit-bearing, instructor-led classes and 
independent learners who do not receive credit or instructor monitoring. Over the past decade, more than four million 
students have enrolled in 40 OL/ courses offered in both high schools and colleges.* 


« Method of Delivery: Courses and learning materials are available on the OL! website. While OL! content is delivered online, 
instructors may choose to supplement the online material with face-to-face reinforcement (hybrid delivery). Two studies 
featured in this report (Bowen et al., 2014; Lovett, Meyer, & Thille, 2008) investigated the effects of OL/ that used a blended 
approach, which supplemented online instruction provided through OLI/ with in-person instruction. One study (Schunn & 
Patchan, 2009) investigated the effects of OL/ using online instruction only. 


« Frequency and Duration of Service: OL! provides content that ranges in length from Comparison Group: In the 
several-hour modules to full-semester courses. Independent learners may complete three studies that contribute 
the material at their own pace, while students in instructor-led courses may be to this Intervention Report, 
assigned to complete the material in a specified timeframe. students in the comparison 


group completed 
coursework in traditional 
face-to-face settings. 


* Intervention Components: The OL/ model includes several components, which are 
noted in Table 2. 
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Table 2. Components of OL! 


Key component OLI 

Learning objectives Every OL! course, section, and page lists learning objectives, allowing students to assess whether they have understood key 
concepts. 

Expository materials Materials include readings, examples, and multi-media components that support learning objectives. These expository 
materials can be downloaded from OL/to create a textbook. 

Learning activities with Two types of embedded learning activities provide opportunities for students to practice what they have learned and test their 

immediate feedback understanding: 


+ Lean by Doing activities include multiple choice questions, simulations, matching, and other options to help students 
practice a new skill or understand a new concept. 

+ Did Get This? activities are self-assessments that pose questions to students to see if they understood a specific concept. 
These activities are presented after Learn by Doing activities. 

Students receive immediate feedback after each response addressing the misconception behind an incorrect answer or 

reinforcing the reasoning behind a correct answer. Checkpoint quizzes are also available within OL/, which provide detailed 

scores and information for instructors, and can be used to calculate course grades. 


Learning Dashboard The Learning Dashboard provides instructors with real-time student-level information about progress through OL/ course 
activities, interaction patterns with course materials, and class-level accuracy in answering questions related to each learning 
objective. 

Gradebook Student performance on quizzes is reported in the leaming dashboard as well as the gradebook. In the gradebook, instructors can 


view grades, adjust grades, change grading options, and grant exceptions to individual students. 


Face-to-face reinforcement — Some instructors supplement the online OL/ material with one to two hours of face-to-face instruction every week. 
(optional) 


What Does OL! Cost? 


The WWC identified a number of cost components from the OL! website. The cost breakdown below is not designed to be 
exhaustive; rather, it is designed to provide educators an overview of the kinds of resources needed to implement OLI. 


¢ Personnel Costs: OL! can be delivered as part of an academic course at a college. OL! does not incur any personnel costs 
beyond those normally associated with teaching a course. 


¢ Facilities Costs: OL/ courses delivered in a hybrid format require physical space to accommodate periodic face-to-face 
instruction. 


« Equipment and Materials Costs: All OL! courses require internet access and an up-to-date web browser. Some content 
cannot be accessed on mobile devices. 


¢ Costs Paid by Students or Parents: OL/ courses may be completed by independent learners or by students in credit- 
bearing, instructor-led classes. Most of the OL/ courses are free to independent learners, but a few charge a $10 
maintenance fee. Most OL/ courses delivered for credit will charge students a maintenance fee of $25, with fees ranging 
from $10 to $80 per student. Course fees may be paid by individual students or by colleges. Students are also subject to any 
tuition fees assessed by their college. 


¢ In-Kind Supports: OL/ does not include any in-kind supports. 


« Sources of Funding: The William and Flora Hewlett Foundation funded the development of the first four OL! courses 
at Carnegie Mellon University in 2002. Further development of OL/ courses was supported by the Bill & Melinda Gates 
Foundation, the Kresge Foundation, the Lumina Foundation, the National Science Foundation, the Spencer Foundation, and 
the Walter S. Johnson Foundation. Ongoing funding and operational support are provided by Carnegie Mellon University. 
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For More Information: 
About OL/ 
The Simon Initiative, Carnegie Mellon University 
5000 Forbes Avenue, Pittsburgh, PA 15213 
Web: http://oli.cmu.edu/. Phone: (412) 268-3294 


For detailed cost information, please see: http://oli.cmu.edu/olis-payment-options-and-cost/. 


About the studies reviewed 
Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. |. (2014). Interactive learning online at public universities: Evidence from a six-campus randomized 
trial. Journal of Policy Analysis and Management, 33(1), 94-111. Retrieved from https://eric.ed.gov/?id=EJ1027704 


Lovett, M., Meyer, 0., & Thille, C. (2008). The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student 
learning. Journal of Interactive Media in Education, 1. Retrieved from https://eric.ed.gov/?id=EJ840810 


Schunn, C. D., & Patchan, M. (2009). An evaluation of accelerated learning in the CMU Open Learning Initiative course “Logic & Proofs” Pittsburgh, PA: 
University of Pittsburgh, Learning Research and Development Center. 


In What Context Was OL! Studied? 


The following section provides information on the setting and participants involved in the three studies of OL/ that meet WWC 
evidence standards. This information can help educators understand the context in which the studies of OL! were conducted 
so that they can better determine whether the program might be suitable for their setting. 


WHERE THE STUDY WAS CONDUCTED 


3 studies, 942 students in 8 colleges and universities in Maryland, New York, and Pennsylvania 


1 private 4-year college, 7 public 4-year colleges 


10 11-12 @ 


Postsecondary (PS) 


LEARN MORE 


Read more about the OL/intervention and the studies that are summarized here on the Intervention Report webpage. 


ENDNOTES 

' For more information, please see Recommendation 2 in the WWC Practice Guide, Using Technology to Support Postsecondary Student 
Learning. 

* The descriptive information for this intervention comes from OLI's website (https://oli.cmu.edu/) and from Bowen et al. (2014), Lovett, 
Meyer, & Thille (2008), and Schunn & Patchan (2009). The What Works Clearinghouse (WWC) requests developers review the intervention 
description sections for accuracy from their perspective. The WWC provided the developer with the intervention description in March 
2019 and the WWC incorporated feedback from the developer. Further verification of the accuracy of the descriptive information for this 
intervention is beyond the scope of this review. 

3 The literature search reflects documents publicly available by March 2019. Reviews of the studies in this report used the standards from 
the WWC Procedures and Standards Handbook (version 3.0) and the Supporting Postsecondary Success review protocol (version 3.0). 
The evidence presented in this report is based on available research. Findings and conclusions could change as new research becomes 
available. 

* Bier, N., Moore, S., & Van Velsen, M. (2019). Instrumenting courseware and leveraging data with the Open Learning Initiative (OLI). 
Companion Proceedings 9th International Learning Analytics & Knowledge Conference, Tempe, AZ. 
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